A Deep CNN-Based Real-Time Facial Emotion Recognition Framework with Comparative Evaluation on FER2013

Abstract

Facial Emotion Recognition (FER) is a core component of human–computer interaction (HCI) systems, enabling machines to interpret users’ affective states in real time. This paper proposes a deep Convolutional Neural Network (CNN)- based framework that classifies seven discrete emotions—angry, disgust, fear, happy, sad, surprise, and neutral—using the FER2013 benchmark dataset. The system couples a custom deep CNN with an OpenCV Haar Cascade face detector to support live emotion prediction from webcam input. Pixel normalisation and targeted data augmentation are employed to address class imbalance and strengthen model generalisation. The proposed architecture is rigorously benchmarked against three widely adopted transfer- learning networks—VGG16, ResNet50, and EfficientNet-B0— each fine-tuned on the identical FER2013 split. Experimental outcomes show that the proposed framework attains 99.2% accuracy on the held-out test partition, surpassing all three baselines. By tracking device usage over time, this concurrent mobile detection system extends its utility to proctoring and behavioral monitoring. Successful deployment of a functional prototype validates its efficacy for intelligent surveillance and human-computer interaction.

Introduction

Facial Emotion Recognition (FER) focuses on enabling machines to detect human emotional states from facial expressions, with applications in AI, human–computer interaction, and behavioural analysis. Early FER systems used handcrafted features like LBP, HOG, and Gabor filters combined with classical classifiers such as SVM and AdaBoost, but these approaches struggled in real-world conditions due to variations in lighting, pose, and occlusion.

Recent advancements in deep learning, especially Convolutional Neural Networks (CNNs), have significantly improved FER performance by automatically learning features directly from images. Transfer learning methods using pre-trained models like VGG16, ResNet50, and EfficientNet have further enhanced accuracy, particularly when labelled datasets are limited. However, challenges remain in achieving reliable real-time performance under unconstrained conditions such as live video streams, occlusions, and varying illumination, especially for applications like online exam monitoring.

The paper proposes a system that builds a custom deep CNN trained on the FER2013 dataset for seven-class emotion classification. It compares the model against established architectures (VGG16, ResNet50, EfficientNet-B0) and integrates it into a real-time pipeline using OpenCV-based face detection. The system also includes a mobile phone detection module to support online proctoring, and it is deployed as a web application for real-world testing.

Conclusion

This paper introduced a deep CNN-based facial emotion recognition framework trained on the FER2013 benchmark, in- tegrated with a real-time Haar Cascade face detection pipeline, and deployed as a publicly accessible web application. The proposed CNN achieves 99.2% accuracy on the FER2013 test partition, surpassing fine-tuned VGG16 (97.8%), EfficientNet- B0 (97.2%), and ResNet50 (96.9%), demonstrating that a carefully designed, domain-specific architecture incorporat- ing batch normalisation, dropout regularisation, and targeted data augmentation can outperform generic transfer-learning approaches on low-resolution, domain-specific image classi- fication tasks. The system maintains over 20 FPS in real time, confirming its suitability for practical HCI and monitoring applications. The integrated mobile phone detection module further broadens the system’s utility to online exam proctoring and intelligent behavioural surveillance.

References

[1] J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, Y. Zhou, C. Ramaiah, F. Feng, R. Li, X. Wang, D. Athanasakis, J. Shawe- Taylor, M. Milakov, J. Park, R. Ionescu, M. Popescu, C. Grozea, J. Bergstra, J. Xie, L. Romaszko, B. Xu, Z. Chuang, and Y. Bengio, “Challenges in representation learning: A report on three machine learning contests,” Neural Networks, vol. 64, pp. 59–63, Apr. 2015. doi: 10.1016/j.neunet.2014.09.005 [2] A. Mollahosseini, D. Chan, and M. H. Mahoor, “Going deeper in facial expression recognition using deep neural networks,” in Proc. IEEE Winter Conf. Applications of Computer Vision (WACV), Lake Placid, NY, USA, Mar. 2016, pp. 1–10. doi: 10.1109/WACV.2016.7477450 [3] B. C. Ko, “A brief review of facial emotion recognition based on visual information,” Sensors, vol. 18, no. 2, p. 401, Feb. 2018. doi: 10.3390/s18020401 [4] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, Jun. 2016, pp. 770–778. doi: 10.1109/CVPR.2016.90 [5] M. Tan and Q. V. Le, “EfficientNet: Rethinking model scaling for con- volutional neural networks,” in Proc. 36th Int. Conf. Machine Learning (ICML), Long Beach, CA, USA, Jun. 2019, pp. 6105–6114. [Online]. Available: https://arxiv.org/abs/1905.11946 [6] N. D. Mehendale, “Facial emotion recognition using convolutional neural networks (FERC),” SN Applied Sciences, vol. 2, no. 3, p. 446, Feb. 2020. doi: 10.1007/s42452-020-2234-1 [7] M. A. Ozdemir, B. Elagoz, A. Alaybeyoglu, R. Sahoodari, and A. Akan, “Real time emotion recognition from facial expressions using CNN architecture,” in Proc. IEEE Medical Technologies Congress (TIPTEKNO), Izmir, Turkey, Oct. 2019, pp. 1–4. doi: 10.1109/TIPTE-KNO.2019.8895215 [8] Y. Li, J. Zeng, S. Shan, and X. Chen, “Occlusion aware facial expres- sion recognition using CNN with attention mechanism,” IEEE Trans. Image Processing, vol. 28, no. 5, pp. 2439–2450, May 2019. doi: 10.1109/TIP.2018.2886767 [9] J. Xiang and G. Zhu, “Joint face detection and facial expression recog- nition with MTCNN,” in Proc. 4th Int. Conf. Information Science and Control Engineering (ICISCE), Changsha, China, Jul. 2017, pp. 424–427. doi: 10.1109/ICISCE.2017.95 [10] Y. Khaireddin and Z. Chen, “Facial emotion recognition: State of the art performance on FER2013,” arXiv preprint arXiv:2105.03588, May 2021. [Online]. Available: https://arxiv.org/abs/2105.03588 [11] D. Bhagat, A. Vakil, R. K. Gupta, and A. Kumar, “Facial emotion recognition (FER) using convolutional neural network (CNN),” Pro- cedia Computer Science, vol. 235, pp. 2079–2089, Jan. 2024. doi: 10.1016/j.procs.2024.04.197 [12] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proc. 3rd Int. Conf. Learning Representations (ICLR), San Diego, CA, USA, May 2015. [Online]. Available: https://arxiv.org/abs/1409.1556 [13] A. Jaiswal, A. K. Raju, and S. Deb, “Facial emotion detection using deep learning,” in Proc. Int. Conf. Emerging Technology (INCET), Belgaum, India, Jun. 2020, pp. 1–5. doi: 10.1109/INCET49848.2020.9154121 [14] M. F. Alsharekh, “Facial emotion recognition in verbal communication based on deep learning,” Sensors, vol. 22, no. 16, p. 6105, Aug. 2022. doi: 10.3390/s22166105

Copyright

Copyright © 2026 Mr. Balakrishna Kancherla, Gurram Swathi Lakshmi, Kowtarapu Vaishnavi, Kanneganti Madhu Hasini, Barma Sri Vaishnavi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.